{
 "cells": [
  {
   "metadata": {},
   "cell_type": "markdown",
   "source": [
    "# 增强 RAG 系统的重排\n",
    "\n",
    "实现重排序技术以提高 RAG 系统中的检索质量。重排序作为初始检索后的第二步过滤，确保使用最相关的内容进行响应生成。\n",
    "\n",
    "------\n",
    "重排序的核心概念\n",
    "\n",
    "1. **初始检索**：第一轮使用基础相似性搜索（速度较快但准确性较低）\n",
    "2. **文档评分**：评估每个检索到的文档与查询的相关性\n",
    "3. **重新排序**：按相关性分数对文档进行排序\n",
    "4. **选择**：仅使用最相关的文档进行响应生成\n",
    "\n",
    "------\n",
    "实现步骤：\n",
    "- 处理文档以创建向量存储：从PDF 中提取文本，分割文本块并创建向量存储\n",
    "- 创建查询嵌入并检索文档：初步查询，需要召回较多文档\n",
    "- 应用重排：\n",
    "    - 基于 LLM 的重排序：使用 LLM 相关性评分对搜索结果进行重排序\n",
    "    - 基于关键词的重排序：关键词匹配的次数和关键词所在位置的简单重排序方法\n",
    "- 利用重排序组合成的上下文生成回答"
   ],
   "id": "cbd3b253158d0e7"
  },
  {
   "metadata": {
    "ExecuteTime": {
     "end_time": "2025-04-24T10:03:46.559843Z",
     "start_time": "2025-04-24T10:03:45.179666Z"
    }
   },
   "cell_type": "code",
   "source": [
    "import fitz\n",
    "import os\n",
    "import re\n",
    "import json\n",
    "import numpy as np\n",
    "from tqdm import tqdm\n",
    "from openai import OpenAI\n",
    "from dotenv import load_dotenv\n",
    "\n",
    "load_dotenv()"
   ],
   "id": "be892da1c4f43437",
   "outputs": [
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 1,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "execution_count": 1
  },
  {
   "metadata": {
    "ExecuteTime": {
     "end_time": "2025-04-24T10:03:49.623411Z",
     "start_time": "2025-04-24T10:03:49.295161Z"
    }
   },
   "cell_type": "code",
   "source": [
    "client = OpenAI(\n",
    "    base_url=os.getenv(\"LLM_BASE_URL\"),\n",
    "    api_key=os.getenv(\"LLM_API_KEY\")\n",
    ")\n",
    "llm_model = os.getenv(\"LLM_MODEL_ID\")\n",
    "embedding_model = os.getenv(\"EMBEDDING_MODEL_ID\")\n",
    "\n",
    "pdf_path = \"../../data/AI_Information.en.zh-CN.pdf\""
   ],
   "id": "290a21716c81e85f",
   "outputs": [],
   "execution_count": 2
  },
  {
   "metadata": {
    "ExecuteTime": {
     "end_time": "2025-04-24T10:03:52.449901Z",
     "start_time": "2025-04-24T10:03:52.441170Z"
    }
   },
   "cell_type": "code",
   "source": [
    "def extract_text_from_pdf(pdf_path):\n",
    "    \"\"\"\n",
    "    从 PDF 文件中提取文本，并打印前 `num_chars` 个字符。\n",
    "\n",
    "    Args:\n",
    "    pdf_path (str): Path to the PDF file.\n",
    "\n",
    "    Returns:\n",
    "    str: Extracted text from the PDF.\n",
    "    \"\"\"\n",
    "    # 打开 PDF 文件\n",
    "    mypdf = fitz.open(pdf_path)\n",
    "    all_text = \"\"  # 初始化一个空字符串以存储提取的文本\n",
    "\n",
    "    # Iterate through each page in the PDF\n",
    "    for page_num in range(mypdf.page_count):\n",
    "        page = mypdf[page_num]\n",
    "        text = page.get_text(\"text\")  # 从页面中提取文本\n",
    "        all_text += text  # 将提取的文本追加到 all_text 字符串中\n",
    "\n",
    "    return all_text  # 返回提取的文本"
   ],
   "id": "5dcd0a094e4a3267",
   "outputs": [],
   "execution_count": 3
  },
  {
   "metadata": {
    "ExecuteTime": {
     "end_time": "2025-04-24T10:03:54.098200Z",
     "start_time": "2025-04-24T10:03:54.091216Z"
    }
   },
   "cell_type": "code",
   "source": [
    "def chunk_text(text, n, overlap):\n",
    "    \"\"\"\n",
    "    将文本分割为重叠的块\n",
    "\n",
    "    Args:\n",
    "    text (str): 要分割的文本\n",
    "    n (int): 每个块的字符数\n",
    "    overlap (int): 块之间的重叠字符数\n",
    "\n",
    "    Returns:\n",
    "    List[str]: 文本块列表\n",
    "    \"\"\"\n",
    "    chunks = []  #\n",
    "    for i in range(0, len(text), n - overlap):\n",
    "        # 添加从当前索引到索引 + 块大小的文本块\n",
    "        chunks.append(text[i:i + n])\n",
    "\n",
    "    return chunks  # Return the list of text chunks"
   ],
   "id": "beae42f71b3b30e0",
   "outputs": [],
   "execution_count": 4
  },
  {
   "metadata": {
    "ExecuteTime": {
     "end_time": "2025-04-24T10:03:55.369225Z",
     "start_time": "2025-04-24T10:03:55.360370Z"
    }
   },
   "cell_type": "code",
   "source": [
    "class SimpleVectorStore:\n",
    "    \"\"\"\n",
    "    使用NumPy实现的简单向量存储。\n",
    "    \"\"\"\n",
    "    def __init__(self):\n",
    "        \"\"\"\n",
    "        初始化向量存储。\n",
    "        \"\"\"\n",
    "        self.vectors = []  # 用于存储嵌入向量的列表\n",
    "        self.texts = []  # 用于存储原始文本的列表\n",
    "        self.metadata = []  # 用于存储每个文本元数据的列表\n",
    "\n",
    "    def add_item(self, text, embedding, metadata=None):\n",
    "        \"\"\"\n",
    "        向向量存储中添加一个项目。\n",
    "\n",
    "        Args:\n",
    "        text (str): 原始文本。\n",
    "        embedding (List[float]): 嵌入向量。\n",
    "        metadata (dict, 可选): 额外的元数据。\n",
    "        \"\"\"\n",
    "        self.vectors.append(np.array(embedding))  # 将嵌入转换为numpy数组并添加到向量列表中\n",
    "        self.texts.append(text)  # 将原始文本添加到文本列表中\n",
    "        self.metadata.append(metadata or {})  # 添加元数据到元数据列表中，如果没有提供则使用空字典\n",
    "\n",
    "    def similarity_search(self, query_embedding, k=5):\n",
    "        \"\"\"\n",
    "        查找与查询嵌入最相似的项目。\n",
    "\n",
    "        Args:\n",
    "        query_embedding (List[float]): 查询嵌入向量。\n",
    "        k (int): 返回的结果数量。\n",
    "\n",
    "        Returns:\n",
    "        List[Dict]: 包含文本和元数据的前k个最相似项。\n",
    "        \"\"\"\n",
    "        if not self.vectors:\n",
    "            return []  # 如果没有存储向量，则返回空列表\n",
    "\n",
    "        # 将查询嵌入转换为numpy数组\n",
    "        query_vector = np.array(query_embedding)\n",
    "\n",
    "        # 使用余弦相似度计算相似度\n",
    "        similarities = []\n",
    "        for i, vector in enumerate(self.vectors):\n",
    "            # 计算查询向量与存储向量之间的余弦相似度\n",
    "            similarity = np.dot(query_vector, vector) / (np.linalg.norm(query_vector) * np.linalg.norm(vector))\n",
    "            similarities.append((i, similarity))  # 添加索引和相似度分数\n",
    "\n",
    "        # 按相似度排序（降序）\n",
    "        similarities.sort(key=lambda x: x[1], reverse=True)\n",
    "\n",
    "        # 返回前k个结果\n",
    "        results = []\n",
    "        for i in range(min(k, len(similarities))):\n",
    "            idx, score = similarities[i]\n",
    "            results.append({\n",
    "                \"text\": self.texts[idx],  # 添加对应的文本\n",
    "                \"metadata\": self.metadata[idx],  # 添加对应的元数据\n",
    "                \"similarity\": score  # 添加相似度分数\n",
    "            })\n",
    "\n",
    "        return results  # 返回前k个最相似项的列表\n"
   ],
   "id": "4dbca32386374e92",
   "outputs": [],
   "execution_count": 5
  },
  {
   "metadata": {
    "ExecuteTime": {
     "end_time": "2025-04-24T10:03:57.380794Z",
     "start_time": "2025-04-24T10:03:57.376524Z"
    }
   },
   "cell_type": "code",
   "source": [
    "def create_embeddings(text):\n",
    "    \"\"\"\n",
    "    使用Embedding模型为给定文本创建嵌入向量。\n",
    "\n",
    "    Args:\n",
    "    text (str): 要创建嵌入向量的输入文本。\n",
    "\n",
    "    Returns:\n",
    "    List[float]: 嵌入向量。\n",
    "    \"\"\"\n",
    "    # 通过将字符串输入转换为列表来处理字符串和列表输入\n",
    "    input_text = text if isinstance(text, list) else [text]\n",
    "\n",
    "    # 使用指定的模型为输入文本创建嵌入向量\n",
    "    response = client.embeddings.create(\n",
    "        model=embedding_model,\n",
    "        input=input_text\n",
    "    )\n",
    "\n",
    "    # 如果输入是字符串，仅返回第一个嵌入向量\n",
    "    if isinstance(text, str):\n",
    "        return response.data[0].embedding\n",
    "\n",
    "    # 否则，将所有嵌入向量作为向量列表返回\n",
    "    return [item.embedding for item in response.data]"
   ],
   "id": "106ec9706b2f2982",
   "outputs": [],
   "execution_count": 6
  },
  {
   "metadata": {
    "ExecuteTime": {
     "end_time": "2025-04-24T10:03:58.488577Z",
     "start_time": "2025-04-24T10:03:58.480012Z"
    }
   },
   "cell_type": "code",
   "source": [
    "def process_document(pdf_path, chunk_size=1000, chunk_overlap=200):\n",
    "    \"\"\"\n",
    "    为RAG处理文档。\n",
    "\n",
    "    Args:\n",
    "    pdf_path (str): PDF文件的路径。\n",
    "    chunk_size (int): 每个文本块的大小（以字符为单位）。\n",
    "    chunk_overlap (int): 文本块之间的重叠大小（以字符为单位）。\n",
    "\n",
    "    Returns:\n",
    "    SimpleVectorStore: 包含文档文本块及其嵌入向量的向量存储。\n",
    "    \"\"\"\n",
    "    print(\"从PDF中提取文本...\")\n",
    "    extracted_text = extract_text_from_pdf(pdf_path)  # 调用函数提取PDF中的文本\n",
    "\n",
    "    print(\"分割文本...\")\n",
    "    chunks = chunk_text(extracted_text, chunk_size, chunk_overlap)  # 将提取的文本分割为多个块\n",
    "    print(f\"创建了 {len(chunks)} 个文本块\")\n",
    "\n",
    "    print(\"为文本块创建嵌入向量...\")\n",
    "    # 为了提高效率，一次性为所有文本块创建嵌入向量\n",
    "    chunk_embeddings = create_embeddings(chunks)\n",
    "\n",
    "    # 创建向量存储\n",
    "    store = SimpleVectorStore()\n",
    "\n",
    "    # 将文本块添加到向量存储中\n",
    "    for i, (chunk, embedding) in enumerate(zip(chunks, chunk_embeddings)):\n",
    "        store.add_item(\n",
    "            text=chunk,  # 文本内容\n",
    "            embedding=embedding,  # 嵌入向量\n",
    "            metadata={\"index\": i, \"source\": pdf_path}  # 元数据，包括索引和源文件路径\n",
    "        )\n",
    "\n",
    "    print(f\"向向量存储中添加了 {len(chunks)} 个文本块\")\n",
    "    return store\n"
   ],
   "id": "f79a807e5b4881ef",
   "outputs": [],
   "execution_count": 7
  },
  {
   "metadata": {},
   "cell_type": "markdown",
   "source": [
    "## 实现基于 LLM 的重排序\n",
    "实现基于 LLM 的重排序函数。"
   ],
   "id": "bcbb79d3c2f8805a"
  },
  {
   "metadata": {
    "ExecuteTime": {
     "end_time": "2025-04-24T10:04:02.262854Z",
     "start_time": "2025-04-24T10:04:02.255239Z"
    }
   },
   "cell_type": "code",
   "source": [
    "def rerank_with_llm(query, results, top_n=3):\n",
    "    \"\"\"\n",
    "    使用 LLM 相关性评分对搜索结果进行重排序。\n",
    "\n",
    "    Args:\n",
    "        query (str): 用户查询\n",
    "        results (List[Dict]): 初始搜索结果\n",
    "        top_n (int): 重排序后要返回的结果数量\n",
    "\n",
    "    Returns:\n",
    "        List[Dict]: 重排序后的结果\n",
    "    \"\"\"\n",
    "    print(f\"正在重排序 {len(results)} 个文档...\")  # 打印要重排序的文档数量\n",
    "\n",
    "    scored_results = []  # 初始化一个空列表以存储评分结果\n",
    "\n",
    "    # 定义 LLM 的系统提示\n",
    "    system_prompt = \"\"\"\n",
    "    您是文档相关性评估专家，擅长判断文档与搜索查询的匹配程度。您的任务是根据文档对给定查询的应答质量，给出0到10分的评分。\n",
    "\n",
    "    评分标准：\n",
    "    0-2分：文档完全无关\n",
    "    3-5分：文档含部分相关信息但未直接回答问题\n",
    "    6-8分：文档相关且能部分解答查询\n",
    "    9-10分：文档高度相关且直接准确回答问题\n",
    "\n",
    "    必须仅返回0到10之间的单个整数评分，不要包含任何其他内容。\n",
    "    \"\"\"\n",
    "\n",
    "    # 遍历每个结果\n",
    "    for i, result in enumerate(results):\n",
    "        # 每 5 个文档显示一次进度\n",
    "        if i % 5 == 0:\n",
    "            print(f\"正在评分文档 {i+1}/{len(results)}...\")\n",
    "\n",
    "        # 定义 LLM 的用户提示\n",
    "        user_prompt = f\"\"\"\n",
    "        查询: {query}\n",
    "\n",
    "        文档:\n",
    "        {result['text']}\n",
    "\n",
    "        请对文档的相关性进行评分，评分范围为 0 到 10, 并仅返回一个整数。\n",
    "        \"\"\"\n",
    "\n",
    "        # 获取 LLM 的响应\n",
    "        response = client.chat.completions.create(\n",
    "            model=llm_model,\n",
    "            temperature=0,\n",
    "            messages=[\n",
    "                {\"role\": \"system\", \"content\": system_prompt},\n",
    "                {\"role\": \"user\", \"content\": user_prompt}\n",
    "            ]\n",
    "        )\n",
    "\n",
    "        # 从 LLM 响应中提取评分\n",
    "        score_text = response.choices[0].message.content.strip()\n",
    "\n",
    "        # 使用正则表达式提取数值评分\n",
    "        score_match = re.search(r'\\b(10|[0-9])\\b', score_text)\n",
    "        if score_match:\n",
    "            score = float(score_match.group(1))\n",
    "        else:\n",
    "            # 如果评分提取失败，使用相似度评分作为备选\n",
    "            print(f\"警告：无法从响应中提取评分：'{score_text}'，使用相似度评分代替\")\n",
    "            score = result[\"similarity\"] * 10\n",
    "\n",
    "        # 将评分结果添加到列表中\n",
    "        scored_results.append({\n",
    "            \"text\": result[\"text\"],\n",
    "            \"metadata\": result[\"metadata\"],\n",
    "            \"similarity\": result[\"similarity\"],\n",
    "            \"relevance_score\": score\n",
    "        })\n",
    "\n",
    "    # 按相关性评分降序对结果进行排序\n",
    "    reranked_results = sorted(scored_results, key=lambda x: x[\"relevance_score\"], reverse=True)\n",
    "\n",
    "    # 返回前 top_n 个结果\n",
    "    return reranked_results[:top_n]\n"
   ],
   "id": "b2a51add18c0238e",
   "outputs": [],
   "execution_count": 8
  },
  {
   "metadata": {},
   "cell_type": "markdown",
   "source": "## 基于简单关键词的重排序",
   "id": "f32b89fd6ebcb459"
  },
  {
   "metadata": {
    "ExecuteTime": {
     "end_time": "2025-04-24T10:04:04.714039Z",
     "start_time": "2025-04-24T10:04:04.705654Z"
    }
   },
   "cell_type": "code",
   "source": [
    "def rerank_with_keywords(query, results, top_n=3):\n",
    "    \"\"\"\n",
    "    基于关键词匹配和位置的简单重排序方法。\n",
    "\n",
    "    Args:\n",
    "        query (str): 用户查询\n",
    "        results (List[Dict]): 初始搜索结果\n",
    "        top_n (int): 重排序后返回的结果数量\n",
    "\n",
    "    Returns:\n",
    "        List[Dict]: 重排序后的结果\n",
    "    \"\"\"\n",
    "    # 从查询中提取重要关键词\n",
    "    keywords = [word.lower() for word in query.split() if len(word) > 3]\n",
    "\n",
    "    scored_results = []  # 初始化一个列表以存储评分结果\n",
    "\n",
    "    for result in results:\n",
    "        document_text = result[\"text\"].lower()  # 将文档文本转换为小写\n",
    "\n",
    "        # 基础分数从向量相似度开始\n",
    "        base_score = result[\"similarity\"] * 0.5\n",
    "\n",
    "        # 初始化关键词分数\n",
    "        keyword_score = 0\n",
    "        for keyword in keywords:\n",
    "            if keyword in document_text:\n",
    "                # 每找到一个关键词加一些分数\n",
    "                keyword_score += 0.1\n",
    "\n",
    "                # 如果关键词出现在文本开头部分，额外加分\n",
    "                first_position = document_text.find(keyword)\n",
    "                if first_position < len(document_text) / 4:  # 在文本的前四分之一部分\n",
    "                    keyword_score += 0.1\n",
    "\n",
    "                # 根据关键词出现的频率加分\n",
    "                frequency = document_text.count(keyword)\n",
    "                keyword_score += min(0.05 * frequency, 0.2)  # 最大值限制为 0.2\n",
    "\n",
    "        # 通过结合基础分数和关键词分数计算最终得分\n",
    "        final_score = base_score + keyword_score\n",
    "\n",
    "        # 将评分结果添加到列表中\n",
    "        scored_results.append({\n",
    "            \"text\": result[\"text\"],\n",
    "            \"metadata\": result[\"metadata\"],\n",
    "            \"similarity\": result[\"similarity\"],\n",
    "            \"relevance_score\": final_score\n",
    "        })\n",
    "\n",
    "    # 按最终相关性分数降序对结果进行排序\n",
    "    reranked_results = sorted(scored_results, key=lambda x: x[\"relevance_score\"], reverse=True)\n",
    "\n",
    "    # 返回前 top_n 个结果\n",
    "    return reranked_results[:top_n]\n"
   ],
   "id": "43f94ce18275ddf3",
   "outputs": [],
   "execution_count": 9
  },
  {
   "metadata": {},
   "cell_type": "markdown",
   "source": "## 回答生成",
   "id": "c16d74c4deafbfae"
  },
  {
   "metadata": {
    "ExecuteTime": {
     "end_time": "2025-04-24T10:04:09.385044Z",
     "start_time": "2025-04-24T10:04:09.379120Z"
    }
   },
   "cell_type": "code",
   "source": [
    "def generate_response(query, context):\n",
    "    \"\"\"\n",
    "    根据查询和上下文生成响应。\n",
    "\n",
    "    Args:\n",
    "        query (str): 用户查询\n",
    "        context (str): 获取到的上下文\n",
    "\n",
    "    Returns:\n",
    "        str: 生成的响应\n",
    "    \"\"\"\n",
    "    # 定义系统提示以指导AI的行为\n",
    "    system_prompt = \"您是一个乐于助人的AI助手。请仅根据提供的上下文来回答用户的问题。如果在上下文中找不到答案，请直接说'没有足够的信息'。\"\n",
    "\n",
    "    # 通过组合上下文和查询创建用户提示\n",
    "    user_prompt = f\"\"\"\n",
    "        上下文:\n",
    "        {context}\n",
    "\n",
    "        问题: {query}\n",
    "\n",
    "        请基于上述上下文内容提供一个全面详尽的答案。\n",
    "    \"\"\"\n",
    "\n",
    "    # 使用指定的模型生成响应\n",
    "    response = client.chat.completions.create(\n",
    "        model=llm_model,\n",
    "        temperature=0,\n",
    "        messages=[\n",
    "            {\"role\": \"system\", \"content\": system_prompt},\n",
    "            {\"role\": \"user\", \"content\": user_prompt}\n",
    "        ]\n",
    "    )\n",
    "\n",
    "    # 返回生成的响应内容\n",
    "    return response.choices[0].message.content\n"
   ],
   "id": "cead17eecbbd114a",
   "outputs": [],
   "execution_count": 10
  },
  {
   "metadata": {},
   "cell_type": "markdown",
   "source": [
    "### 具有重排序功能的完整RAG管道\n",
    "\n",
    "到目前为止，我们已经实现了RAG管道的核心组件，包括文档处理、问题回答和重排序。现在，我们将把这些组件结合起来，创建一个完整的RAG管道。"
   ],
   "id": "ba17bc561f5d845"
  },
  {
   "metadata": {
    "ExecuteTime": {
     "end_time": "2025-04-24T10:04:13.894836Z",
     "start_time": "2025-04-24T10:04:13.888351Z"
    }
   },
   "cell_type": "code",
   "source": [
    "def rag_with_reranking(query, vector_store, reranking_method=\"llm\", top_n=3):\n",
    "    \"\"\"\n",
    "    完整的RAG管道，包含重排序功能。\n",
    "\n",
    "    Args:\n",
    "        query (str): 用户查询\n",
    "        vector_store (SimpleVectorStore): 向量存储\n",
    "        reranking_method (str): 重排序方法 ('llm' 或 'keywords')\n",
    "        top_n (int): 重排序后返回的结果数量\n",
    "\n",
    "    Returns:\n",
    "        Dict: 包括查询、上下文和响应的结果\n",
    "    \"\"\"\n",
    "    # 创建查询嵌入\n",
    "    query_embedding = create_embeddings(query)\n",
    "\n",
    "    # 初步检索（获取比重排序所需更多的结果）\n",
    "    initial_results = vector_store.similarity_search(query_embedding, k=10)\n",
    "\n",
    "    # 应用重排序\n",
    "    if reranking_method == \"llm\":\n",
    "        reranked_results = rerank_with_llm(query, initial_results, top_n=top_n)\n",
    "    elif reranking_method == \"keywords\":\n",
    "        reranked_results = rerank_with_keywords(query, initial_results, top_n=top_n)\n",
    "    else:\n",
    "        # 不进行重排序，直接使用初步检索的前几个结果\n",
    "        reranked_results = initial_results[:top_n]\n",
    "\n",
    "    # 将重排序结果的上下文合并\n",
    "    context = \"\\n\\n===\\n\\n\".join([result[\"text\"] for result in reranked_results])\n",
    "\n",
    "    # 根据上下文生成响应\n",
    "    response = generate_response(query, context)\n",
    "\n",
    "    return {\n",
    "        \"query\": query,\n",
    "        \"reranking_method\": reranking_method,\n",
    "        \"initial_results\": initial_results[:top_n],\n",
    "        \"reranked_results\": reranked_results,\n",
    "        \"context\": context,\n",
    "        \"response\": response\n",
    "    }\n"
   ],
   "id": "b42d525372e8d2e4",
   "outputs": [],
   "execution_count": 11
  },
  {
   "metadata": {},
   "cell_type": "markdown",
   "source": "## 评估重排质量",
   "id": "2744d72d68509ec1"
  },
  {
   "metadata": {
    "ExecuteTime": {
     "end_time": "2025-04-24T10:04:20.429783Z",
     "start_time": "2025-04-24T10:04:20.425146Z"
    }
   },
   "cell_type": "code",
   "source": [
    "# Load the validation data from a JSON file\n",
    "with open('../../data/val.json', 'r', encoding='utf-8') as f:\n",
    "    data = json.load(f)\n",
    "\n",
    "# Extract the first query from the validation data\n",
    "query = data[0]['question']\n",
    "\n",
    "# Extract the reference answer from the validation data\n",
    "reference_answer = data[0]['ideal_answer']\n"
   ],
   "id": "e82fd14d6532c006",
   "outputs": [],
   "execution_count": 12
  },
  {
   "metadata": {
    "ExecuteTime": {
     "end_time": "2025-04-24T10:05:58.892077Z",
     "start_time": "2025-04-24T10:04:23.150438Z"
    }
   },
   "cell_type": "code",
   "source": [
    "# 处理文档\n",
    "vector_store = process_document(pdf_path)  # 将PDF文件中的文本提取、分块并生成向量存储\n",
    "\n",
    "# 示例查询\n",
    "query = \"人工智能是否有可能改变我们的生活方式和工作方式？\"  # 查询示例\n",
    "\n",
    "# 比较不同的检索方法\n",
    "print(\"比较检索方法...\")  # 打印提示信息，表示开始比较不同方法的检索效果\n",
    "\n",
    "# 1. 标准检索（无重排序）\n",
    "print(\"\\n=== 标准检索 ===\")  # 打印标题，表示当前使用的是标准检索方法\n",
    "standard_results = rag_with_reranking(query, vector_store, reranking_method=\"none\")  # 使用标准检索方法进行检索\n",
    "print(f\"\\nQuery: {query}\")  # 打印查询内容\n",
    "print(f\"\\nResponse:\\n{standard_results['response']}\")  # 打印标准检索方法生成的响应\n",
    "\n",
    "# 2. 基于LLM的重排序\n",
    "print(\"\\n=== 基于LLM的重排序 ===\")  # 打印标题，表示当前使用的是基于LLM的重排序方法\n",
    "llm_results = rag_with_reranking(query, vector_store, reranking_method=\"llm\")  # 使用基于LLM的重排序方法进行检索\n",
    "print(f\"\\nQuery: {query}\")  # 打印查询内容\n",
    "print(f\"\\nResponse:\\n{llm_results['response']}\")  # 打印基于LLM的重排序方法生成的响应\n",
    "\n",
    "# 3. 基于关键词的重排序\n",
    "print(\"\\n=== 基于关键词的重排序 ===\")  # 打印标题，表示当前使用的是基于关键词的重排序方法\n",
    "keyword_results = rag_with_reranking(query, vector_store, reranking_method=\"keywords\")  # 使用基于关键词的重排序方法进行检索\n",
    "print(f\"\\nQuery: {query}\")  # 打印查询内容\n",
    "print(f\"\\nResponse:\\n{keyword_results['response']}\")  # 打印基于关键词的重排序方法生成的响应\n"
   ],
   "id": "7bbf58cb611fb34b",
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "从PDF中提取文本...\n",
      "分割文本...\n",
      "创建了 13 个文本块\n",
      "为文本块创建嵌入向量...\n",
      "向向量存储中添加了 13 个文本块\n",
      "比较检索方法...\n",
      "\n",
      "=== 标准检索 ===\n",
      "\n",
      "Query: 人工智能是否有可能改变我们的生活方式和工作方式？\n",
      "\n",
      "Response:\n",
      "根据提供的上下文，人工智能（AI）确实有可能深刻改变我们的生活方式和工作方式，具体体现在以下多个方面：\n",
      "\n",
      "---\n",
      "\n",
      "### **1. 工作方式的变革**\n",
      "- **自动化与效率提升**  \n",
      "  - **制造业**：AI驱动的工业机器人（如协作机器人）已广泛应用于焊接、装配等任务，提升精度和效率，同时与人类协同工作。  \n",
      "  - **服务业**：AI通过客服聊天机器人、自动化库存管理等减少重复性劳动，例如零售业的个性化推荐和供应链优化。  \n",
      "  - **岗位转型**：虽然AI可能导致部分岗位流失（如常规性任务），但也将创造新职业（如AI系统监管、伦理审查），需通过再培训应对转型。\n",
      "\n",
      "- **决策支持**  \n",
      "  AI工具可分析海量数据，辅助人类在医疗诊断、金融预测等领域做出更精准的决策，例如通过医疗影像分析提高诊断效率。\n",
      "\n",
      "---\n",
      "\n",
      "### **2. 生活方式的革新**\n",
      "- **个性化服务**  \n",
      "  - **教育**：AI提供自适应学习平台和虚拟导师，根据学生需求定制内容，如自动评分和个性化反馈。  \n",
      "  - **娱乐**：AI算法推荐电影、音乐或生成艺术（如AI绘画、作曲），丰富文化消费体验。  \n",
      "  - **零售**：通过分析用户数据，AI提供定制化购物建议，优化消费体验。\n",
      "\n",
      "- **健康与医疗**  \n",
      "  - **个性化医疗**：AI分析患者数据以预测疾病风险并制定治疗方案，如加速药物研发和机器人辅助手术。  \n",
      "  - **远程医疗**：AI驱动的诊断工具可提升偏远地区的医疗可及性。\n",
      "\n",
      "- **交通与出行**  \n",
      "  自动驾驶技术依赖AI感知环境并决策，可能减少交通事故并改变通勤模式。\n",
      "\n",
      "---\n",
      "\n",
      "### **3. 社会与伦理影响**\n",
      "- **伦理挑战**  \n",
      "  - **偏见与公平**：AI可能放大训练数据中的偏见，需通过可解释AI（XAI）增强透明度。  \n",
      "  - **隐私问题**：AI依赖大量数据，需严格监管以确保隐私安全（如边缘AI的本地数据处理）。  \n",
      "  - **自主武器**：AI武器化引发安全担忧，需国际法规约束。\n",
      "\n",
      "- **协作模式**  \n",
      "  未来人机协作将更紧密，例如AI增强人类创造力（辅助设计、科研）或体力劳动（物流机器人），而非完全取代人类。\n",
      "\n",
      "---\n",
      "\n",
      "### **4. 未来趋势**\n",
      "- **技术融合**  \n",
      "  - **量子计算**：可能加速AI算法，推动材料科学、药物研发等领域的突破。  \n",
      "  - **边缘AI**：设备端数据处理减少延迟，增强隐私（如智能家居、物联网）。  \n",
      "\n",
      "- **社会福祉**  \n",
      "  AI被用于应对气候变化、贫困等全球性问题，例如优化资源分配或监测环境变化。\n",
      "\n",
      "---\n",
      "\n",
      "### **结论**  \n",
      "AI将通过自动化、个性化服务和跨领域融合重塑工作和生活方式，但其发展需伴随负责任的治理（如伦理框架、国际标准）以平衡创新与风险。持续的技术进步与社会适应将决定AI变革的最终方向。\n",
      "\n",
      "=== 基于LLM的重排序 ===\n",
      "正在重排序 10 个文档...\n",
      "正在评分文档 1/10...\n",
      "正在评分文档 6/10...\n",
      "\n",
      "Query: 人工智能是否有可能改变我们的生活方式和工作方式？\n",
      "\n",
      "Response:\n",
      "是的，人工智能（AI）已经并将在未来继续深刻改变我们的生活方式和工作方式。以下从多个领域展开具体分析：\n",
      "\n",
      "### **一、生活方式的改变**\n",
      "1. **交通运输**  \n",
      "   - 自动驾驶汽车通过AI感知环境并决策，将提高道路安全性、减少拥堵，并可能改变私家车所有权模式（如共享出行）。\n",
      "\n",
      "2. **零售与消费**  \n",
      "   - AI驱动的个性化推荐、库存管理和聊天机器人优化购物体验，例如根据用户数据预测需求并提供定制优惠。\n",
      "\n",
      "3. **医疗保健**  \n",
      "   - **诊断与治疗**：AI分析医学影像（如X光片）可提高诊断准确性；个性化医疗通过患者数据预测最佳治疗方案。  \n",
      "   - **药物研发**：AI加速药物筛选过程，缩短新药上市时间。  \n",
      "   - **机器人手术**：AI辅助系统提升手术精度，减少人为误差。\n",
      "\n",
      "4. **教育**  \n",
      "   - **个性化学习**：AI平台根据学生进度调整内容，例如提供定制化习题或反馈。  \n",
      "   - **虚拟导师**：AI助手解答问题并跟踪学习效果，扩大教育资源覆盖范围。  \n",
      "   - **自动评分**：节省教师时间，同时提供即时反馈。\n",
      "\n",
      "5. **娱乐与创意**  \n",
      "   - AI生成艺术、音乐和文学内容（如绘画、作曲），辅助人类创作者或独立产出作品。  \n",
      "   - 游戏和虚拟现实中，AI增强用户交互体验，例如动态调整剧情或难度。\n",
      "\n",
      "6. **隐私与安全**  \n",
      "   - AI在网络安全中检测威胁并自动响应，但同时也引发数据隐私担忧（如个人信息被滥用）。\n",
      "\n",
      "---\n",
      "\n",
      "### **二、工作方式的变革**\n",
      "1. **行业应用**  \n",
      "   - **制造业**：AI用于预测性维护（提前发现设备故障）、质量控制和流程优化，降低成本并提升效率。  \n",
      "   - **金融**：AI自动化欺诈检测、风险评估和算法交易，提高数据处理速度。  \n",
      "   - **人力资源**：AI筛选简历、分析员工绩效，甚至设计培训计划。\n",
      "\n",
      "2. **自动化与就业影响**  \n",
      "   - **岗位替代**：重复性任务（如流水线作业、客服）可能被AI取代，但新角色（如AI伦理师、数据科学家）将涌现。  \n",
      "   - **人机协作**：AI辅助人类决策，例如医生结合AI诊断结果制定治疗方案。\n",
      "\n",
      "3. **技能需求变化**  \n",
      "   - 未来职场需掌握与AI协作的技能（如数据分析、AI工具操作），推动再培训和教育改革。\n",
      "\n",
      "4. **伦理挑战**  \n",
      "   - **偏见问题**：AI可能放大训练数据中的歧视（如招聘算法偏向特定群体）。  \n",
      "   - **透明度**：“黑箱”决策机制（如深度学习）需提高可解释性以建立信任。\n",
      "\n",
      "---\n",
      "\n",
      "### **三、未来趋势与挑战**\n",
      "- **可解释AI（XAI）**：推动AI决策透明化，解决伦理争议。  \n",
      "- **自主系统**：需制定法规控制AI武器化等风险。  \n",
      "- **社会适应**：应对自动化导致的失业问题，需政策干预（如基本收入制度）。\n",
      "\n",
      "### **结论**  \n",
      "AI正全方位重塑生活和工作的各个层面，带来效率提升和便利性，但也伴随隐私、就业和伦理等挑战。其最终影响取决于技术发展与社会治理的平衡。\n",
      "\n",
      "=== 基于关键词的重排序 ===\n",
      "\n",
      "Query: 人工智能是否有可能改变我们的生活方式和工作方式？\n",
      "\n",
      "Response:\n",
      "是的，人工智能（AI）有潜力显著改变我们的生活方式和工作方式。根据上下文，以下是AI可能带来的具体影响：\n",
      "\n",
      "### 1. **工作方式的变革**\n",
      "- **自动化与效率提升**：AI在制造业、物流和零售等行业中用于自动化任务（如装配、库存管理），提高效率并降低成本。这可能导致某些重复性岗位的流失，但同时会创造新的技术相关职位。\n",
      "- **人机协作**：未来工作将更依赖人类与AI系统的协作，例如AI工具辅助决策、增强生产力或提供实时数据分析支持。\n",
      "- **教育与技能需求**：AI驱动的个性化学习和虚拟导师将改变教育方式，要求劳动力持续学习新技能以适应技术发展。\n",
      "\n",
      "### 2. **生活方式的转变**\n",
      "- **个性化服务**：在零售、娱乐和医疗领域，AI通过分析用户数据提供个性化推荐（如购物、音乐）、定制化医疗方案或自适应学习体验。\n",
      "- **智能家居与交通**：边缘AI技术使设备本地处理数据，提升智能家居响应速度；自动驾驶汽车将改变出行模式。\n",
      "- **健康管理**：AI在医疗诊断、药物研发和机器人手术中的应用，可能延长寿命并提高生活质量。\n",
      "\n",
      "### 3. **社会与伦理影响**\n",
      "- **隐私与安全**：AI依赖大量数据，需平衡便利性与隐私保护（如通过边缘AI增强本地数据处理）。\n",
      "- **公平与偏见**：需解决AI系统潜在的偏见问题，确保决策透明（如可解释AI/XAI的发展）。\n",
      "- **社会挑战应对**：AI可用于气候变化、贫困等全球性问题，但需国际协作治理（如防止AI武器化）。\n",
      "\n",
      "### 4. **未来趋势**\n",
      "- **新兴技术融合**：量子计算可能加速AI算法，推动科学突破；AI与机器人技术结合将扩展至制造业、医疗等场景。\n",
      "- **创造力与创新**：AI生成艺术、音乐和内容的能力可能重塑文化创作方式，辅助人类创新。\n",
      "\n",
      "### 结论\n",
      "AI的进步将深刻重构工作流程（自动化、新职业）、生活方式（个性化、智能化）及社会结构（伦理、治理）。其影响取决于如何通过“负责任的开发”和“国际监管”引导技术向造福社会的方向发展（如“AI造福社会”倡议）。持续的技术适应与伦理框架建设是关键。\n"
     ]
    }
   ],
   "execution_count": 13
  },
  {
   "metadata": {
    "ExecuteTime": {
     "end_time": "2025-04-24T10:06:40.079913Z",
     "start_time": "2025-04-24T10:06:40.074153Z"
    }
   },
   "cell_type": "code",
   "source": [
    "def evaluate_reranking(query, standard_results, reranked_results, reference_answer=None):\n",
    "    \"\"\"\n",
    "    评估重排序结果与标准结果的质量。\n",
    "\n",
    "    Args:\n",
    "        query (str): 用户查询\n",
    "        standard_results (Dict): 标准检索方法的结果\n",
    "        reranked_results (Dict): 重排序检索方法的结果\n",
    "        reference_answer (str, 可选): 用于比较的参考答案\n",
    "\n",
    "    Returns:\n",
    "        str: 评估输出\n",
    "    \"\"\"\n",
    "    # 定义AI评估系统的提示信息\n",
    "    system_prompt = \"\"\"\n",
    "    您是RAG系统评估专家，负责对比分析两种检索方法的效果。请比较两种检索方法获取的上下文及生成的回答，\n",
    "    评估哪种方法能提供更优质的上下文和更精准、全面的回答。\n",
    "    \"\"\"\n",
    "\n",
    "\n",
    "    # 准备带有截断上下文和响应的对比文本\n",
    "    comparison_text = f\"\"\"查询内容: {query}\n",
    "\n",
    "标准检索上下文:\n",
    "{standard_results['context'][:1000]}... [已截断]\n",
    "\n",
    "标准检索回答:\n",
    "{standard_results['response']}\n",
    "\n",
    "重排序检索上下文:\n",
    "{reranked_results['context'][:1000]}... [已截断]\n",
    "\n",
    "重排序检索回答:\n",
    "{reranked_results['response']}\"\"\"\n",
    "\n",
    "    # 如果提供了参考答案，则将其包含在对比文本中\n",
    "    if reference_answer:\n",
    "        comparison_text += f\"\"\"\n",
    "\n",
    "参考答案:\n",
    "{reference_answer}\"\"\"\n",
    "\n",
    "    # 创建AI评估员的用户提示\n",
    "    user_prompt = f\"\"\"\n",
    "{comparison_text}\n",
    "\n",
    "请从以下维度评估检索方法表现：\n",
    "1. 上下文相关性\n",
    "2. 回答准确性\n",
    "3. 回答全面性\n",
    "4. 整体性能\n",
    "\n",
    "要求提供包含具体示例的详细分析报告。\n",
    "\"\"\"\n",
    "\n",
    "    # 使用指定模型生成评估响应\n",
    "    response = client.chat.completions.create(\n",
    "        model=llm_model,\n",
    "        temperature=0,\n",
    "        messages=[\n",
    "            {\"role\": \"system\", \"content\": system_prompt},\n",
    "            {\"role\": \"user\", \"content\": user_prompt}\n",
    "        ]\n",
    "    )\n",
    "\n",
    "    # 返回评估输出\n",
    "    return response.choices[0].message.content\n"
   ],
   "id": "7c8cb0d2248a3473",
   "outputs": [],
   "execution_count": 14
  },
  {
   "metadata": {
    "ExecuteTime": {
     "end_time": "2025-04-24T10:07:28.914645Z",
     "start_time": "2025-04-24T10:06:44.698825Z"
    }
   },
   "cell_type": "code",
   "source": [
    "# 评估重排序结果与标准结果的质量\n",
    "evaluation = evaluate_reranking(\n",
    "    query=query,  # 用户查询\n",
    "    standard_results=standard_results,  # 标准检索方法的结果\n",
    "    reranked_results=llm_results,  # 基于LLM重排序方法的结果\n",
    "    reference_answer=reference_answer  # 用于比较的参考答案\n",
    ")\n",
    "\n",
    "# 打印评估结果\n",
    "print(\"\\n=== 评估结果 ===\")\n",
    "print(evaluation)\n"
   ],
   "id": "9fa6431a0a3a674b",
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "=== 评估结果 ===\n",
      "### **RAG系统评估报告：标准检索 vs. 重排序检索**\n",
      "\n",
      "#### **1. 上下文相关性**\n",
      "- **标准检索**  \n",
      "  - **优点**：上下文覆盖了AI对工作方式（如制造业、服务业）和生活方式（如医疗、教育）的广泛影响，并包含伦理讨论（如偏见、隐私）和未来趋势（如量子计算）。  \n",
      "  - **缺点**：部分内容与查询的直接关联较弱（如AI武器化），且信息组织较分散（如第六章的机器人技术与问题关联性低）。  \n",
      "  - **示例**：提到“边缘AI”和“量子计算”虽技术前瞻，但未直接说明如何改变生活/工作方式。\n",
      "\n",
      "- **重排序检索**  \n",
      "  - **优点**：更聚焦于具体领域（如交通、零售、医疗），每个段落直接对应生活或工作的某一方面，逻辑连贯。  \n",
      "  - **缺点**：未来趋势部分（如XAI）被截断，缺少标准检索中的技术融合（如量子计算）内容。  \n",
      "  - **示例**：对“自动驾驶汽车”和“个性化医疗”的描述更直接关联查询，但未提及“人机协作”的深度案例。\n",
      "\n",
      "**评分**：重排序检索 > 标准检索（更贴近问题核心）\n",
      "\n",
      "---\n",
      "\n",
      "#### **2. 回答准确性**\n",
      "- **标准检索回答**  \n",
      "  - **优点**：准确引用上下文，如制造业中协作机器人的应用、AI在医疗诊断中的辅助作用。  \n",
      "  - **缺点**：部分推断超出上下文范围（如“量子计算加速药物研发”未在上下文中明确提及）。  \n",
      "\n",
      "- **重排序检索回答**  \n",
      "  - **优点**：严格基于上下文，如自动驾驶的安全性、零售业个性化推荐，数据支持充分。  \n",
      "  - **缺点**：未发现明显不准确之处，但未涵盖标准检索中的“边缘AI”优势。  \n",
      "\n",
      "**评分**：重排序检索 ≥ 标准检索（更少推测性内容）\n",
      "\n",
      "---\n",
      "\n",
      "#### **3. 回答全面性**\n",
      "- **标准检索回答**  \n",
      "  - **优点**：覆盖领域广，包括工作转型、伦理挑战、未来技术，甚至社会福祉（如应对气候变化）。  \n",
      "  - **缺点**：部分领域（如教育）仅简要提及，未展开具体案例。  \n",
      "\n",
      "- **重排序检索回答**  \n",
      "  - **优点**：对生活（交通、医疗、教育）和工作（制造业、金融）的细分领域描述更详细，如“AI自动评分系统”的具体作用。  \n",
      "  - **缺点**：缺少“社会福祉”和“量子计算”等宏观视角。  \n",
      "\n",
      "**示例对比**：  \n",
      "- 标准检索提到“AI造福社会”但无细节；重排序检索详细说明“AI在医疗影像诊断中的应用”。  \n",
      "- 两者均提到伦理问题，但重排序检索更具体（如招聘算法偏见）。  \n",
      "\n",
      "**评分**：标准检索 > 重排序检索（覆盖更广，但重排序检索在细分领域更深入）\n",
      "\n",
      "---\n",
      "\n",
      "#### **4. 整体性能**\n",
      "- **标准检索**  \n",
      "  - **优势**：宏观视角强，适合需要了解AI全局影响的用户。  \n",
      "  - **劣势**：部分内容冗余，关键信息需用户自行筛选。  \n",
      "\n",
      "- **重排序检索**  \n",
      "  - **优势**：信息密度高，直接回答“如何改变”，适合寻求具体案例的用户。  \n",
      "  - **劣势**：技术前瞻性内容不足。  \n",
      "\n",
      "**改进建议**：  \n",
      "- 标准检索可优化段落排序，优先呈现与问题最相关的内容（如将“制造业”案例提前）。  \n",
      "- 重排序检索可补充未来趋势的上下文（如XAI的完整定义）。  \n",
      "\n",
      "**最终结论**：  \n",
      "- **优先推荐**：**重排序检索**（更精准、结构更清晰）。  \n",
      "- **补充建议**：若用户需要技术前瞻或伦理深度讨论，可结合标准检索的部分内容。  \n",
      "\n",
      "---\n",
      "\n",
      "### **具体改进示例**\n",
      "1. **标准检索优化**：  \n",
      "   - 将“人机协作”部分与“制造业机器人”合并，删除冗余的武器化内容。  \n",
      "   - 添加“量子计算在药物研发中的具体案例”（需扩展上下文）。  \n",
      "\n",
      "2. **重排序检索优化**：  \n",
      "   - 补充“可解释AI（XAI）”完整定义，增强伦理讨论的完整性。  \n",
      "   - 增加“边缘AI”对智能家居的影响案例。  \n",
      "\n",
      "**评分总结**：  \n",
      "| 维度         | 标准检索 | 重排序检索 |  \n",
      "|--------------|----------|------------|  \n",
      "| 上下文相关性 | 7/10     | 9/10       |  \n",
      "| 回答准确性   | 8/10     | 9/10       |  \n",
      "| 回答全面性   | 9/10     | 8/10       |  \n",
      "| **整体性能** | **8/10** | **9/10**   |\n"
     ]
    }
   ],
   "execution_count": 15
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}
